Sparse sampling heuristic search

In the article A Sparse Sampling Algorithm for Near-Optimal Planning in Large Markov Decision Processes, Kearns et al show that there’re now theoratical boundary’s to solve an mdp with an infinite state space in a such a way that the running time has no dependency’s on the size of the statespace. To come to this conclusion they introduce an algorithm that uses a sparse look-a-head tree to select a near optimal action. The algorithm is however exponentional in the horizon time (which depends on the discount factor λ and the desired aproximation), in there article they leave as an open question the possibility that there might be an algorithm which given a generative model (a very common kind of simulator) can come up with an near-optimal solution. In this report whe show that such an algorithm can’t use the hoeffding-bound to estimate his chances of being wrong. Another problem for practical implementation is that altough the amount of samples needed may not depend on the size of the state-space it still can get very high, we therefore have combined the original algorithm of Kearns et al. with a heuristic wich points out were to sample next based on the results of the samples already taken. Altough in the worst case we need to take exactly the amount of sample Kearns et al. need, it’s quite easy to think of mdp’s in wich this heurisitc will save you from taking a lot of not needed samples without loosing any certainty on your aproximation.